In last week's episode of The Cockroach Hour, Jim Walker chatted with Red Hat principal product manager Scott McCarty to talk about everything from the what the future of serverless and distroless are, to what happens when you run Oracle on Kubernetes. The full video is available here, and the transcript is below.
Jim Walker:
Welcome, everybody. Welcome to our event today, The Cockroach Hour. Today, we have a special guest from Red Hat joining us, so we're going to talk about OpenShift. We're going to talk about containers, we're going to talk about Kubernetes, all this goodness. And I'm going to presume we're going to end up talking a little bit about open source too, after meeting our presenter here. But let me just give a little quick guidance before we get started here. We do have a QA panel, there is a chat panel as well. Please engage however you like, we enjoy questions on The Cockroach Hour.
So with that, thank you all for joining and today's session is... We've been asked before, can you tell if these things are basic, intermediate or advanced? I think of this session as more of a basic conversation. We're not going to get into any command line. We're not going to get too deep into distributed transactions like we have in the past.
This is a little bit more higher level stuff, workloads, Kubernetes, definitions of these things, where people are getting tripped up on these things and talking through those sort of things, right? But please do ask questions, we will send mugs out to people. I have a mug around here somewhere, I model it. So with that, let's bring the cameras on. So Tim and Scott, if you guys want to join me. There you are Tim and then Scott.
Scott McCarty:
Hi, Jim.
Jim Walker:
So everybody in the crowd, it's guys in flannel shirts today.
Scott McCarty:
Flannel shirt Wednesday.
Jim Walker:
Our outfits are boring but I promise that the conversation will be lively and we'll keep it to some really interesting topics here. But first and foremost, thank you, Scott, for joining us. Scott McCarty is a principal product manager in charge of containers at Red Hat. So, Scott, what do you work on at Red Hat? A title is what? Four words.
Scott McCarty:
Yeah.
Jim Walker:
What are you working on at Red Hat?
Scott McCarty:
But it's all about the syllables. If you're a principal product manager, that's like what? Seven, eight syllables.
Jim Walker:
And then you throw containers in there, there's another couple there.
Scott McCarty:
Yeah. No, three syllables, you're good to go. Yeah, I used to have a title that was 16 syllables. I thought that was way cooler but I used to make fun of myself much more. So in this role, I live in what's called... So there's two main platforms at Red Hat, if you will. There's OpenShift and there's REL and I live in this nexus in between the two, where our team builds technology like CRI-O, Podman, Buildah, Skopeo. We work on runC, we work on all these low level bits. I also manage the roadmap for Red Hat Universal Base Image. So all these primitives are what I call them. They're like the basic flour, sugar, eggs and water of containers that we've basically built for Red Hat.
Jim Walker:
I should have worn my Rocket T-shirt today, knowing what you work on. So anyway, we'll come back to that though, if anybody-
Scott McCarty:
Yeah. We consider Podman and the spiritual successor of Rockets.
Jim Walker:
It is, it is, it is. I know but the little Rocket logo is so great and everything. Scott, how long have you been at Red Hat?
Scott McCarty:
I have been at Red Hat nine years.
Jim Walker:
Yeah, that's awesome.
Scott McCarty:
Yeah, it's been a while. Nine and a half.
Jim Walker:
It's amazing when I meet Red Hat employees. The tenure is long, I've met people, a dozen... Well, over a dozen years. What is it? Red Hat's been around for how long, 20 years now?
Scott McCarty:
On our company call today, Mike Evans passed 25 years. So we've been around since what? Over 25 years.
Jim Walker:
The start date.
Scott McCarty:
I don't remember the start date, '93. I want to say '93 was the start date.
Jim Walker:
So I can finally say, there's some people who have been there a couple dozen years. But to me, one of the most important companies of my generation and all of our generations in terms of, I think all three of us here are open source advocates and zealots and all those. I don't have a neck beard but I definitely love open source. So thank you, Scott, for joining us. And then Tim Vale, if you want to introduce yourself.
Tim Vail:
Well hey, everybody. Tim Vail here, head of solutions engineering at Cockroach Labs, been here about two and a half years. And as Jim mentioned, spent a lot of time in open source prior to that. So glad to be here, glad to be talking about all this fun stuff.
Jim Walker:
And Tim brings in an angle of engagement with customers every day, both large and small. Smaller companies using CockroachDB Dedicated, larger companies trying to deploy CockroachDB on Kubernetes or in these distributed environments, in a single cloud, multicloud, lots of everything. We're going to talk about, hopefully, all those things today. My name is Jim Walker. I'm in product marketing here at Cockroach Labs. I am just a week short of two years at this company. Man, Scott, you're just way beyond me in any sort of tenure but that's okay. I was at CoreOS before this, so I almost could have made it to Red Hat but that's okay.
Jim Walker:
But welcome, everybody. Like we usually start, good morning, good afternoon and good evening. Please do use chat and we'll come back to this. Not thank you, we're not done yet. So I'm going to stop sharing so that it's us on here. So Scott, your title is principal product manager of containers. So when you have to describe what a container is to somebody and let's just say it's the most highest level, what is it?
Scott McCarty:
Yeah, so it depends on who I'm talking to. For this audience, I'd say it's probably more technical audience, so I'd describe it as... Even the simple technical definition I'd say, is it's two things. There's a runtime component and then there's an at rest component. So I jokingly say that containers are fancy files and they're fancy processes.
Jim Walker:
That's right.
Scott McCarty:
At rest, they're just fancy files on disk. They're tarballs that have some metadata wrapped around them that are defined by the Open Containers Initiative. And then at runtime, they're just fancy processes that have extra controls in place constraining them. Before Red Hat acquired CoreOS, I wrote an article that took a spin on their CEO's article about, can you run databases and he asked all these different questions that he gets that are... I don't want to say he said they're dumb but we get all these questions that are remedial. They're like, can I run a database in a container? And I'm like, replace the word container with process and then ask that question again.
Jim Walker:
Right, exactly.
Scott McCarty:
Can I run a database in a process? And you're like well, I don't know any other way that you can run a database. So if you use the right mindset, these questions, they're pretty easy to solve themselves, answer themselves. So I joke they're just fancy files and fancy processes. And then there's one other component, there's a registry server. It's just a fancy file server. And other than that, that's it. Fancy files, fancy processes, fancy file servers.
Jim Walker:
Yeah, nothing really new. There's just so many names spaces that have been around for a long time here, y'all. So this is just something, if anybody understands Linux, it's a pretty easy transition to do this. But Scott, if I think about containers... And your title, right? Is principal product manager for containers. And I know there's container platform and Kubernetes Engine and there's OpenShift. Is there anything to innovate in containers? Or are we at steady state and basically, the OCI has defined this and Docker did a great job of pushing everything? Are we at steady state with containers?
Scott McCarty:
Well for me, no. It's funny because my day in and day out, I'll admit, I'm down in the engine room. I jokingly say, most of the time, these are things that a CIO won't understand what we're doing. They have to just trust that we're constantly adding new features and making it better but they don't really necessarily understand what that is, nor should they need to. That's why you buy something like OpenShift. But that said, no, my days are fairly intense.
Jim Walker:
I know.
Scott McCarty:
There's a ton of features that we're still adding in CRI-O and Podman in particular. I'll give you an example, what we're working on right now is image mounts. So something Docker never thought about and nor should they have needed to, it was new. But you're like oh, I want to fire up one container and then I want to modify another container with that. So can I bind mount in this container and then scan it, analyze it or maybe even modify it? And so we have these two features that we're working on in Podman called overlay mounts and then also image mounts. And so you can imagine, this is the flour and sugar and eggs and water of the cake that is containers. If you really look at the way a container works, it's just a bunch of overlay file system layers and all of them are read only except for the top one. And the top one you write to and then when you save the container, you just add one more layer.
Scott McCarty:
It's actually not nearly as complex as people think. And I have diagrams and graphs, I go into these Container Internals Labs. If you Google Container Internals Labs, you'll find all kinds of deep dive information I give. But in a nutshell, once you learn that that's how containers work, you realize you can do it inside of a container as well. So we're looking at, how can we bind mount a container image into a running image, into a running container and then maybe do a read write layer that's a temp FS? And then when this container goes away, this one goes away too but we can scan it and add things.
Scott McCarty:
There's all these basic use cases where you want two containers to be able to talk to each other in a really deep way through a POS-X file system mount, essentially. And so that's one of the things we're doing right now. We need this for, for example, Anchor and and Twistlock and Aqua Sec and all these security scanners. They want a way to bind mount in an image, analyze it read only, so you can't muck it up. But know what's going on it and then save all that data back to a database somewhere. That's the perfect use case for that. Those kinds of features, we do all the time.
Jim Walker:
And I think it's just one of those things, as we become much more mature with distributed systems. Look, really, who was just doing distributed systems five years ago? Okay, really a handful maybe of companies. And some of them probably weren't doing it very well. You walk around QCon today and wow, there's a lot of people interested in this stuff and actually pushing the bounds of these things. And I think Polvi and what the team did at CoreOS to actually help drive some of these core initiatives... They had Rocket, which was their container runtime and I know there's been competing versions of this but to think of what an operating system needs to look like in a container. I think to me, that's the interesting piece, right? Yeah, these things are going to run on an operating... People don't realize, in that container is an OS, right?
Scott McCarty:
An OS, yeah.
Jim Walker:
Let's shrink the size of your container, right? And so I presume just that Linux level of expertise is where you're living, right Scott?
Scott McCarty:
Yeah, it is and it's weird because there's two competing narratives. One narrative is that OS doesn't matter anymore. But then when you dig into the covers, you're like wait a minute, actually, it matters more in a lot of ways than it ever has. So that's a strange part, you go, how did CoreOS get sold for $250 million if the OS doesn't matter? And how did Red Hat get sold for 34 billion if the OS doesn't matter? There's definitely two competing narratives depending on what your interests are and goals and desires are in the Universe.
Jim Walker:
Yeah. The OS always matters, I don't care what anybody says.
Scott McCarty:
Yeah, even with functions as a service, I don't see how it doesn't matter.
Jim Walker:
Yeah, right?
Scott McCarty:
It's still going to matter.
Jim Walker:
Yeah. Really, serverless, it's still servers and there's still an OS running on those things. And I tell you what, there's a whole lot of optimizations that happen and it has to happen at that layer to actually make serverless a reality, in my opinion. I think we are only scratching the surface of what that world is going to be. And what is it a truly serverless environment? To me, it ends up being something Kubernetes like, if you will, almost, Scott. Where you are communicating at that layer for all those things to happen, right?
Scott McCarty:
Yeah, absolutely. Yeah, there's two sides in my world. There's the container images side and the container host side, they're the two main things that we're going to... Now at the container images side, we're always looking at things like serverless and distroless and things like that. And both of these words bother me, serverless and distroless.
Jim Walker:
I know.
Scott McCarty:
Neither of them are actually without servers or without distros, just because you don't put a package manager in a container image, does not mean that there's not a cadre of human beings going out, that are subject matter experts in all these different pieces of software and packaging them in a way that is consumable. That's still the same business requirement, even if you don't use RPMs or maybe you don't use a manager but somebody has to still go do that work for you. And you need to be able to consume I easily.
Jim Walker:
Serverless is just somebody else's servers. That's what the cloud is, right? It's just somebody else's servers, right? Ultimately.
Scott McCarty:
And I joke, distroless, it's just somebody else's distro. Even if you look at the Google server list, a lot of the different build languages rely on Debian packages. They don't include the package manager and you can't rebuild them but they're still pulling from that dependency tree. So dependency trees still matter, even in 2020.
Jim Walker:
So one other thing that's related to this, I think some people who are newer to containers or aren't familiar with the complexity of... You and I talked a little bit before this about the supply chain and how do we get applications to production? There's also things like Ansible or at CoreOS, we had Quay, right? Can you just describe to me what those tools play in this whole supply chain? Because I think they're actually pretty important for people to understand, right? Because the point is not easy.
Scott McCarty:
Yeah, absolutely. So this is probably a good segue into describing what OpenShift 4 is versus 3. If you look at OpenShift 3, we basically had a Kubernetes distribution, OpenShift. We had Ansible deploying that Kubernetes distribution on RHEL, Red Hat Enterprise Linux, which is a Linux distro. And so you can understand all of the primitives there, right? You're like oh, it's a configuration management system, a Kubernetes distro, a Linux distro and wire it all together, you do upgrades. That's pretty easy to understand, that's probably how the whole world did things. You go back to 2005, I was using a homegrown CF engine that we had written at American Greetings, doing e-cards and I left there in 2005. But I called that Web 1.0. That was the tail end of Web 1.0. E-cards were still making more money than you think, even in 2005.
Jim Walker:
For sure.
Scott McCarty:
We did do distributed systems with CF Engine. We were doing with, a CF Engine equivalent, that we were basically pushing out to a thousand Linux web servers. The business problem there is the same, it's running 75 different services across a 1000 different nodes. If you look at OpenShift 3, we're kind of doing the same as we'd always done, except that we were deploying the Kubernetes distro. So you had one workload that you needed to deploy to all of these with a config management system. And then all the other workloads where the application workloads were deployed with Kubernetes Yaml. You could think you'd hit that Kube API and deploy all of your stuff there. But then when you go to OpenShift 4, we actually kind of extended the API down a layer to the cluster itself. I'll admit, I don't think we, even Red Hat have done a good job of explaining what OpenShift 4 is, but it is a profoundly different way of thinking about Kubernetes.
Scott McCarty:
It treats the cluster itself as ... you treat every object in the cluster, including the nodes as objects in the Kubernetes cluster. It's essentially extending that, ... wait, let me back up and say, config management works on the concept of defined outcome. I want this thing to have user added or whatever, make it an item potent, but do it, don't do it 20 times, but add it, make sure it's there. Kubernetes works on a very different paradigm where it's defined state, actual state and it's constantly ... it's got a timer going, a controller that's basically constantly looking for ... looking at the defined state, looking at the actual state, comparing the two and trying to make them look the same.
Scott McCarty:
So that's a very different paradigm because, config management might run on a deploy and then maybe if you were super brave, you'd run it once a day or once an hour or something like that.
Scott McCarty:
Kubernetes is doing it all the time. All the time at any given times per second or minute, it's looking at the state. Managing the cluster itself as part of that defined state is what OpenShift 4 does. So you manage the nodes, you manage the container engine, you manage everything. So if you look there's this thing called the MCO, is what we call it. And it basically is an operator and an operator is basically just an extension of Kubernetes that takes that defined state, actual state and applies it to other things. So we're basically taking this MCO and applying it to the node and the container engine that's running on that node, which are the two main things you need to configure. Historically we did that with Ansible, but now we're moving it into the cloud. So now it's a single API. You literally hit the Kubernetes API, this rest API, and you can manage everything, from the node to the engine, to the containers that are running on it to the state of config files on the cluster. You're treating the cluster as a single computer now through a single API endpoint.
Scott McCarty:
Those two paradigms are very, very different. So then that changes, not to be super complex, but to answer your question, what roles do these things play? They're different in each of those worlds.
Jim Walker:
That's right. That's right. It all collectively comes out as OpenShift. And the value in OpenShift is this greater simplification is abstraction up away from Kubernetes basically and these kind of crazy configurations and, you know, I don't like to write Yaml. I avoided COBOL early on in my life. I don't like to count spaces. I think it's one of these things that why deal with any of that? I want command line, but I don't, maybe a bit of a UI to manage and to do all these different things, but for me what y'all doing with OpenShift is truly tremendous and it's all in open. It's all open source.
Jim Walker:
So there's kind of this boundary though, between Kubernetes and what Red Hat is doing an OpenShift. It's a very tightly integrated platform of what's happening here. How do you guys choose what happens in say, the Kubernetes community and the stuff that Clayton and that whole team is driving coding away on, and then what the OpenShift is seeing. You're a product manager, so you sit between the two sides. How does that work internally with you all?
Scott McCarty:
This is actually touches on this concept of open source as a supply chain. So I think there are fundamentally two ways that people look at open source. There's sort of the ... let's go three, there's three ways. There's companies that look at technology as a closed core, and I'm borrowing Aquaint's and mine's terminology very specifically there, Joseph Jacks calls it “closed core.” It's the idea that we're going to build everything proprietary and we're going to deliver all value to the customer through this proprietary means. Then there's debate about what does open core mean? I'll fully admit there's an open debate about what does that mean.
Scott McCarty:
Red Hat is at the other end of that spectrum where essentially the vast, vast, vast, vast, 99.9% of our supply chain is open source. I leave room for a long tail there of there's little pieces of glue code and things that exist in our environment, like our build systems and there's things behind the scenes that aren't necessarily public, but you don't need them to run OpenShift at runtime or anything. It's just you need some glue code here and there to make your specific environment work.
Scott McCarty:
So I'd say this becomes a supply chain choice. If you look at OpenShift, it's made up of basically, there's two major pieces of the supply chain. Well, actually Kubernetes is a huge chunk of the supply chain for it. I would say that's like the motor in a car, let's make a car comparison. The motor is Kubernetes. You need an electric motor or gasoline or diesel motor. You have to decide this if you're a product manager. You have to decide which motor meets the needs of my customer. You go my customer wants to pull trailers. So maybe we only have diesel and electric as an option, maybe gas isn't good enough. These are the kinds of decisions that the PM makes. So, then you look at OpenShift, I'd say Kubernetes is the motor. And then you look at all the things that run on top of Kubernetes. First let's start with what's below it, Red Hat Core OS, below it. You look at the supply chain, it's Fedora, then RHEL and RHEL Core OS is basically just a different snapshot and version of the exact same bits are in RHEL.
Scott McCarty:
So the supply chain is the Linux is the motor for that. And there's a whole bunch of thousands of other projects around it, for the door handles and everything else. And then you come down into a distribution like Fedora then down into RHEL and RHEL Core OS. You could start to imagine in your mind, a map of this crazy deep web of all these different projects that go into OpenShift. So the supply chain for OpenShift is basically Kubernetes to Linux upstream, major motors and then downstream you go to OKD. OKD is again, an open source freely usable. You don't have to pay a subscription to use it, sort of interim project. Think of it as almost like a distro like Fedora, where it brings it all together.
Scott McCarty:
And there's OpenShift. So how do we decide what goes where? That's the biggest question, right?
Jim Walker:
Yeah.
Scott McCarty:
This is where I try to talk about it in my article about what is open source product management? Your upstream suppliers should do something different than what you do. If your upstream supplier sells fuel injectors and you sell fuel injectors, you're going to have a problem. You guys are never going to be able to differentiate each other, and you're never going to make money. If the upstream supplier sells a fuel injector and you sell a car, it's a lot easier to differentiate. So you have to look at the use case. The use case for OpenShift is it's a, I always joke, it's a dump truck that can go 200 miles an hour and it carries 10 tons of dirt. That's not something that everybody needs, but if you need a dump truck that goes 200 miles an hour and handles pretty well and carries 10 tons of dirt, that's an interesting use case. It's an advanced use case that basically people that don't want to have to manage that entire supply chain themselves handle, and so we decide based on what the need of the user is essentially.
Jim Walker:
Yeah. Yeah. Let's abstract that exact point up Scott. And so, yes, there's the internal workings of the car and fuel injectors and the engine and the drivetrain and the chassis and all these things, and I think those things are important. I think what OpenShift is doing is pulling all that together. So you get a car.
Scott McCarty:
Yeah. Exactly.
Jim Walker:
Or I'm sorry, in this case, a pickup truck. I'm new to these things, you say it's a pickup truck, how do I work this into my architecture today as an enterprise organization. Let's abstract it up to a whole other layer and what am I using it for? What's the value I'm getting out of OpenShift and Kubernetes, at that layer? Without getting too deep into the weeds, what are the workloads people are using, we'll come to that as the next thing, but what is it? Why do I want to buy this thing ultimately?
Scott McCarty:
To be fair, it's not that the workloads are necessarily different, although they can be. The user of OpenShift wants something that just works out of the box and we'll upgrade for 10 years, whatever, seven years in a row. You're not going to get that value out of the upstream. If you're upgrading Kubernetes every six months, you're going to have a lot of work on your hands. The bottom line is that's just a lot of work for you to do. And that costs money, and that costs engineers and engineers are hard to hire. It's pretty natural where the fit is. We want to make the experience lifecycle, FIPs compliance, security, so government organizations, large enough enterprises that they, they would try to tackle this themselves and realize that they don't want to do it because it costs a lot of money and these are very hard engineers to get a hold of.
Scott McCarty:
It's more based on a business understanding of do I really want to tackle all this myself? Do I want to buy a bucket of parts and build a dump truck, or do I just want to buy a dump truck, take it to Ford and have them service it?
Jim Walker:
Right. And I think that's one of the things that people struggle with with Kubernetes. There's a lot of pieces. It's complex. I have to deal with storage and networking and security and all these different things. And I think what OpenShift does is simplify a lot of that for the end user.
Scott McCarty:
It's a prebuilt solution.
Jim Walker:
Exactly. And having a packaged solution around those things. And we used to do this with, with Hadoop. I mean, could somebody roll their own Hadoop distribution eight years ago or whatever? Yeah sure, but why would you do that? And then you need indemnification, all these things.
One of the other areas that you just touched on very clearly and people are asking why is this important? Back in Core OS, this whole concept of tectonic was how did we do rolling upgrades and how do we automate upgrades of software?
Scott McCarty:
There you go. Upgrades is probably one of the biggest challenges of Kubernetes.
Jim Walker:
That's right. Yeah. Each pod has to be updated and there's different layers that have to be updated in each pod. Then the control plan itself needs to be updated. And I think that's one of those core benefits. Talking about the operational complexity of Kubernetes, well, just understanding of the damn thing took me awhile to figure out. Running it in production is difficult. I think that's where the value for ... then again, I'm not the product manager. See I'm on product marketing. See, this is how we work together buddy.
Scott McCarty:
The funny part is you're like, you would never find a business in the Fortune 2000 that would build their own Linux distro. I mean, maybe, if their business is embedded systems where they're doing some very, very specific things, but you're not going to find anybody building their own Linux distro to run SAP on it. That just doesn't even make any sense whatsoever. I think it's a little bit easier to see with a technology that's a little bit older and more mature because you start to realize you're like, "Oh yeah." The perfect car for a family would be like, mom and dad build the car specifically for the family, with the exact size seats for every family members, the exact safety required. Nobody does that. That's irrational and not efficient.
And so the market provides an approximate solution that's pretty good. That's worth paying for essentially. Mom and dad would rather buy a minivan than go build a minivan that's perfect for their family. This is so obvious with cars. It's so obvious with Linux, but it's still not completely obvious with Kubernetes because it's new and people look at it and they go "Oh, this is strategic to our business. So we want to build it and then add all these things to it, and then maintain over time." And one to three years into that, they go, "Yeah, this was not necessarily what we actually wanted to do. We would actually be better if we were building stuff on top of Kubernetes that actually met our business needs as opposed to building Kubernetes."
Jim Walker:
Right. Exactly. I think that's where people are struggling right now. They get too deep in the reeds. Tim, you see people out there using Kubernetes all the time. What are the workloads you're seeing kind of people shift into Kubernetes? Is it all of them and what does that journey look like for them?
Tim:
Yeah. It's funny. I think Kubernetes still for the folks that we talk to is a bit aspirational. It's on everybody's roadmap and certainly those who are doing it, are doing it at the app layer. Customers are certainly talking about, "Well I'm moving my application workloads, my microservices to Kubernetes type architecture." But, obviously here at Cockroach Labs, what are we selling? We're selling a database and we want to, and can run, CockroachDB on Kubernetes. And so we're starting to have the conversation with people not only using Kubernetes to deploy application servers, but to start to deploy a storage layer. That kind of really, at times, makes people's head kind of explode. "What do you mean I can run a database as we were talking about containers?" "Well, sure. Of course." "Yeah. I can run a database and Kubernetes?" "Absolutely, for all the same reasons that you'd want to run your application in Kubernetes."
Not everybody's there yet, but they're getting there. They're definitely getting there.
Jim Walker:
And I think it's one of these things we're going through a paradigm shift. Literally. It's going to take all of us time and, Scott, you're in it. When you're deep in it, you're just there. Whereas if there's a mass majority of people just don't understand, what is the result of distributed systems for you?
Let me shift a little bit into the data and what that actually means in these systems. The first time I ever saw CockroachDB, and here I am now at this company a few years later, Polvi from CoreOS was up on Brian... I forget his last name. The guy who runs OpenStack. I forget the guy's name. He was the president of the OpenStack consortium or whatever. I forget what it was, but we're at OpenStack Summit. Yeah. They were at OpenStack Summit and they wanted to show off Kubernetes and how you couldn't kill this thing, and what was the distributed application? Well, it was the database. Like, holy cow, I can have this distributed database. Right?
And so I think we're seeing people think about Kubernetes. When it first came out, people were like, "Oh, it's for stateless workloads." I didn't even know what that means. You know what I mean? Are you still hearing a lot of those same kind of concerns from customers, Scott? Like, where am I using this thing? How am I going to use it? Like, that's great. I trust you Red Hat and this OpenShift thing's awesome. I'm sold. Are customers struggling with the workloads and how to instrument and how to make it all happen? I presume that's one of the reasons why OpenShift is here in the marketplace and all that. Right? So...
Scott McCarty:
Absolutely. So one of the most popular talks I've given over the last five, six years has been my migration talks. If you look, in a generation we've gone from bare metal to virtual machines. I mean, we can go back as far as you want, mainframes to mini computers, to Unix boxes, to Linux, to virtual machines, but there's always been three options. Right? There's lift and shift. Can I just take this thing I have and run it on this new thing? And does it give me some benefit? Right? And then there's augment. You're like, well, we'll move part of it and then we'll add some things to it to maybe make it work better. And then there's the, let's just write all this crap from scratch. And obviously, it gets more expensive. You go from lift and shift as the cheapest to rewrite everything.
The CIO does not want to hear that you're rewriting everything. So this is constantly like, there's always going to be a business challenge here. Right? And so, yeah, absolutely not. One of my number one conversations is on what can we put in OpenShift or what will work well?
Jim Walker:
Right, exactly.
Scott McCarty:
What do we run on RHEL and what will work well there? And what should we just leave alone because it's not worth moving in? And these are like, yeah, absolutely. And they're really complex questions depending on the environment they require. A lot of times it didn't result in consulting going in and analyzing a whole portfolio of applications, looking at them and all that. And you're like, "These 32 run here, these 72 run there," you know? And it's not an easy question to answer.
But at a minimum, I mean, the thing I try to guide people is if you could separate the code, the configuration, and the data, whatever that means. With MySQL, you can separate the code, configuration, and data. But the problem is the data is one thing, unless you're using Galera or using something like Cockroach. If you have a single piece of data that's one chunk that's not set up to be distributed essentially to have multiple replicas and things like that.
If it's not cloud native, which we tried to teach people with OpenStack, but I think we just rammed through that wall and just people kept going, like, "I can run it like a virtual machine." Now with containers, I think it's finally we're at a hard place where if you really want to run this in Kubernetes, you need to be able to imagine a couple of things. One, this thing could run anywhere, because Kubernetes can decide where it can run it wherever it wants at any given moment.
And then two, it might restart it a thousand times. So you've got to be able to handle two things, right? Can it move anywhere at any time and restart a thousand times? Apache can handle being restarted a thousand times. An Oracle database, not so much like. You restart an Oracle database a thousand times on a thousand different nodes, that's probably going to create a problem.
Jim Walker:
Good luck.
Scott McCarty:
So something like Cockroach that's designed for that is going to handle that a lot better than something that's not designed to do that.
Jim Walker:
Oh, no. And I think it comes back to the core principles of distributed systems. Right? Build it as a small single atomic unit so it can be deployed the same way everywhere. Right? They're built for scale so they can easily scale them, so they communicate in that way. They will survive at all times. Right?
Jim Walker:
And I think those core principles come back. You mentioned something, Scott, that I found interesting. It was like, look at, I can lift and shift. I'll take the thing and I'll dump it into a container and run it. Right? I can make some small changes to it, augment, augment, and okay, maybe it's distributed at that point, whatever that means.
Scott McCarty:
Yeah. Or maybe we run part of it outside the cluster, part of it in the cluster. That's-
Jim Walker:
Right. Exactly. Or do you just re-architect from the ground up? And here at Cockroach Labs, we chose to re-architect from the ground up. And you just set some off in my mind. And I realized why I joined this company is because if you're going to run a database on Kubernetes, you have to re-envision it. It's not as simple as just running it, mounting a persistent volume storage class and figuring that out. If you try to augment or you try to lift and shift, it's not built for that world. Right? And so we realized that with the database, is this some of these things, like if I'm a consumer of OpenShift or I'm going down the path of Kubernetes, maybe my application is just not built for this new world? Is that a question that organizations should be asking themselves?
Scott McCarty:
Oh, yeah. Absolutely. Absolutely. Because if you force a round peg in a square hole, it's not going to work well, and you're going to end up with more nightmares. Like I said, just cram an Oracle database on Kubernetes and see what happens. It'll work great for a few days, months, years, until one day Kubernetes runs into some problem where it can't schedule it and it restarts it 200 times and then the database gets corrupted. And you just end up with these crazy problems. Right? Because it wasn't designed to run in a distributed way.
Jim Walker:
And then somebody was asking a question was like, what are the challenges to moving a database into Kubernetes? And I think that's exactly it, Scott, right? Like if you're going to run a legacy database, like you want to run MySQL or Postgres in a pod? Okay, great. But you need to understand that pod is ephemeral and it may die. And so even with stateful sets, which is a key concept in Kubernetes, if somebody doesn't know what that is, I would check out stateful sets. It's a very interesting way of actually maintaining some sort of state when a pod dies. Right? The simplest explanation, right?
Scott McCarty:
Yeah.
Jim Walker:
Without that, it was a complete lost cause, but even with that, it's still a problem. Right? Because it-
Scott McCarty:
It still won't handle a hundred restarts. That's the other problem I tried to explain. It's not just about where it lands. It's also about how many times it lands there.
Jim Walker:
Right.
Scott McCarty:
There's time and space. Like I always try to explain to people, there's time and space in Kubernetes, and you have to decide can this thing handle being moved anywhere in the universe a thousand times per second? That's a different problem set. It has to be designed for that.
Scott McCarty:
Now there's things like Apache that can handle that pretty well. You're right. Maybe you have a very small website, a blog. It can probably handle that with just a MySQL database. But you do that at scale, that's probably going to fall down. That's where it starts falling down.
Jim Walker:
Yeah. And really it's choosing the right workloads to run in these environments and where it makes sense. Where are you going to get the efficiencies, and why do you need that efficiency, I think is the biggest question, right? And I think that's where I've always had these conversations about workloads and Kubernetes and OpenShift and all these things. It really comes back to that.
Jim Walker:
Going back, though, and just talking about applications, moving them over and this sort of thing, there's this concept of an operator and a Kubernetes. I remember Brandon Phillips and I having a conversation about operators. Actually, I think Brandon wanted to call them “controller operators.” And Rob was like “controllers.” I'm like, "No, man, just operators." Should people be thinking of that? Is that the conduit to make your app work in Kubernetes, or is that just basically something that the software engineering teams that are building things that are going to be deployed, they should be thinking about? Right? You know what I mean? So is this something that's consumer grade or is it more like the industrial grade supply chain vendors have to think about that?
Scott McCarty:
I think it's both. I think both groups of people have to think about it. I mean, that's probably, again, one of the biggest conversations I've had over the last few years is explaining to people what operators are. The way I've broken it down, and I bumped this off Rob to see if he was cool with it, but if you think about traditional operating environments, operational excellence, what did it take to achieve operational excellence? You deployed an app on a server, and then you had a human being also called a sysadmin, that would make sure those two things work together. And that's how you achieved operational excellence, right?
Scott McCarty:
In this Kubernetes world, you really need a robot sysadmin called an operator that now you deploy the app on the platform, and now it's a cluster treated as, again, an OpenShift for paradigm is that you could treat the cluster as a single computer basically, but you're spanning across a bunch of nodes. So it's the platform, the app, and then a robot sysadmin that you deploy side by side with the app that takes care of the app.
So the operator can do things like back it up, restore it, check on it if it's broken. I talk about if it restarts 200 times and the tables get corrupt, the operator will know what to do, right? The operator's what helps handle, coddle these sort of, I call them cloud immigrants. They might not be cloud native, but they've immigrated into the cloud.
Jim Walker:
That's right.
Scott McCarty:
And so coddling that. That's what the robot sysadmin/operator does. And so the knowledge now moves from, we used to have that in a runbook in a Wiki, and instead of in a Wiki, we now have moved it into the operator. Right?
Jim Walker:
Well, yeah, and for a while there-
Scott McCarty:
Somebody has to do that work now.
Jim Walker:
That's right. And for a while there, we moved it into the SRE's head. Right? And so the SRE was basically going off and building scripts. And SRE being one of these artifacts of Google and their whole approach, again, it's like, is this GIFE, which is Google infrastructure for everybody? Right? Like, yeah, kind of. I think the operator was codifying-
Scott McCarty:
I think so, too.
Jim Walker:
Some of the things that the SRE was doing, because operating these things at scale are incredibly difficult. How do you actually manage certificates in a distributed environment across every pod and make sure that everything's going to be sorted out? Okay, Vault's going to probably solve one problem for you, but you still need to distribute and make it... These things are complex.
Scott McCarty:
It's about velocity in my mind. So if you looked at the velocity of the standard application, you could fix it once a day, once a month, once a year. I had a server that we booted up, put RHEL, I forget what it was, six or seven on it, and it ran, probably five or six, sorry, and it ran for 10,000 days or whatever it ran, and then we shut it off. It was ridiculous. It booted once and it shut down once. So we never had to reboot it. There has never been a remotely exploitable kernel bug, so we never had to reboot it. This is what people don't understand.
Scott McCarty:
That kind of velocity is very different than the velocity of, I might not need to do this multiple times per second or minute. And so when you looked at the way config management works in a traditional environment, if, for example, your monitoring system notices a web server goes down, your config manager could go out or you literally could have Nagios talk to Ansible to go out and restart the web server. Right? This would happen once a month, once a week. That's fine. That feedback loop is slow enough, and there's enough slack. It's kind of like a 1960s car. It's like the gear, you find a gear and you can feel it grab. And you're like this car can still go-
Jim Walker:
But that's what makes it fun, man. That's fun. Right? You feel it.
Scott McCarty:
It is fun. But when you're talking about inside the Kubernetes cluster, this defined state, actual state methodology, it's an OODA loop that's basically happening all the time. And so the idea is you're deploying the operator in this tight OODA loop. You're now in the reactor, right? It needs to build a happen and react at the state of the Kubernetes database, so at the speed of the Kubernetes database state change. Right? And so that's what that operator does.
Scott McCarty:
You deploy the automation inside the cluster and it has access to the state data, unlike you. So historically, you had a monitoring system, a fault monitoring system, a config management system, and then the underlying platform and all of these different disparate systems had to talk to each other, and they didn't share a database. But if you deploy all this in Kubernetes, it's all saved in the SED database. And I think in my mind that velocity that that can achieve of restarting a container within seconds of it failing is a different animal than doing it. Oh, well it took us two, three, four minutes to get this thing back up. Okay. No problem back in the day. But at scale with capacity, that doesn't work.
Jim Walker:
Yeah, and it's interesting. You talk about state and how you use that in the context of the operator is funny. When I first joined Cockroach Labs, we got to build an operator. And I remember guys on our team that understand Kubernetes, he's like, "What are you talking about? We don't need an operator." And actually, because we were designed as a distributed system that can actually natively survive a failure of a pod or region or whatever, because you can do that, you don't need a lot of those core operator things that some applications may need. Because you got to think about scale or resilience, right? Scale is how do you deploy and actually get all these things to work, right? Or rolling upgrades. These sorts of things are not always easy to deal with.
How do you apply patches across the huge one?
Scott McCarty:
Yeah, rolling upgrades is a huge one.
Jim Walker:
Yeah. How do you apply patches? And I think those are the core patterns that I see in operators. And we have on operator. I know we're retainer certified and then our operator is now certified by Red Hat as well. I'm probably pushing out some news before Cube, but what the heck? The people on the thing will know. Right?
Jim Walker:
But yeah, we did it because, well, there's certain things or certain deployment patterns that actually are difficult to do, and it goes beyond just being a distributed system. And I think that's the kind of stuff that I think people-
Scott McCarty:
And upgrades-
Jim Walker:
Yeah, sorry.
Scott McCarty:
Are a perfect example.
Jim Walker:
Yeah.
Scott McCarty:
I was going to say, you really struck me when you say upgrades. Upgrades is probably the hardest thing. Again, go back to a traditional environment. Imagine upgrading a MySQL database from, I don't know, a major version, three to four, whatever. And you go, "That's a lot of interaction." You've got to shut the database down, back it, up, run some kind of up, install the new version of the software before you do that.
Scott McCarty:
How do you do that? Which order do you do these things in? You have to shut it down, export the database then shut it down, then remove the software, add the new version, or maybe install them side by side. Then you've got to run some upgrades script that changes a schema. Then you've got to fire the new one up. Oh, it didn't work. Oh, shit. Now how do we get the schema back to the old version to restart this? That's not easy.
Jim Walker:
Scott, that's one instance. That's a stovepipe database.
Scott McCarty:
That's one instance, yeah.
Jim Walker:
Start talking about I have 50 containers all running this software-
Scott McCarty:
That's why you had maintenance windows that were eight hours long.
And SSH into each one of the servers and start figuring this out and do it. So the upgrade process, applying patches, which patches are so incredibly important today. I mean, security problems are just massively important.
Like, you can imagine historically this has happened with go and fiddle-fart with all that figured out. Then we got to automation where it had like an Ansible, or Chef, or CF Engine. We could kind of codify those rules but it still took... I mean, dude Ansible upgraded with something, it still takes hours sometimes to upgrade things. You don't have hours, right? You've got to go figure all that out at the factory and then deploy it in production. I jokingly say containers are about building at the factory, not at the dock, right? Historically you go back to the 1800's, we'd take all the lamps, and pianos, and barrels, and boxes and crates, and literally on carts with horses. We take them down to the dock and load them on and it took freaking three, four or five days. It would take months to load a ship. People don't understand this, like a month. Then we ended up getting the containers and you're like, we loaded the same amount of stuff that's just as complex because we did it at the factory where it was air sealed and clean...and we did it in saltwater with open air. Just put it on there, turn some twist ties, lock the thing down.
Jim Walker:
Right.
Scott McCarty:
That's the idea of an operator. The operator is able to handle just these final last mile operations, with access to the state data of what's going on in the entire system at the same time. But since you have that unified state data, it's like a robot sysadmin that has access to the-
Jim Walker:
That's right.
Scott McCarty:
...it's like I just hooked my brain into a distributed system and like, "Oh, I can feel the state of all the things." It's the matrix. Right? You didn't have that kind of power with a regular configuration management system, but you do with an operator.
Jim Walker:
Right.
Scott McCarty:
Because you can check all these things in real time and figure out what's going on.
Jim Walker:
Right. Honestly, if I think back about a couple of years ago it wasn't that Kubernetes couldn't do this, it's just that Kubernetes is its own layer. It has to do what it does, which is basically let me manage state. Ultimately Kubernetes is one thing, it listens to etcd and says, "This is the state that I need." That's it. Literally if you're going to bring it down to the most simple thing etcd is just \[crosstalk 00:46:57] it says, "Here's my state." As you said, Scott, "I need to make sure that the state looks like this." It means five instances of this, six instances of that, 120 instances of this please, and that's all it does. It's really all it does ultimately in the end right?
Scott McCarty:
Yep.
Jim Walker:
There's a lot of complexity and there's networking and storage and all these things that have to happen. Right?
Scott McCarty:
It handles defining the state really well, not-
Jim Walker:
managing the state-
Scott McCarty:
... changing the state.
Jim Walker:
Right. Exactly.
Scott McCarty:
When there's things you got to do in between the state change.
Jim Walker:
That's right.
Scott McCarty:
Yeah.
Jim Walker:
If people think about operators and where they fit, that's how they fit. There's the thing that's basically controlling and making... it is the Uber robot. then you got a robot controlling the robot. Right? It's robot SRE is what this is, so.
Scott McCarty:
That's what it is, it's a robot SRE. Then OpenShift 4, is basically about having robot SREs for both the applications, which is what regular operators do. Then we've also written system operators that manage the host and the engine, and then all the other software that runs an OpenShift.
Jim Walker:
Let me do my job and make my robot overlords happy as well. Let's talk about the Red Hat Marketplace and what you guys are doing there. I think people are asking a lot of questions about like complexity of applications. Are we just basically building things because engineers have free cycles? Or what is the value of the marketplace? We are definitely in the marketplace and we're seeing adoption through there,. What is the vision and what's going on with OpenShift from that point of view?
Scott McCarty:
So in my opinion, the marketplace is like the final culmination of what we've wanted for 20 years. If you go back, when I first started doing this like '98, there wasn't really a concept of cloud, but automation was already like... Even in 2000, 2001, we knew what automation... Unit sysadmins knew what automation was. Right?
Jim Walker:
Right.
Scott McCarty:
But the problem is everybody built automation for their specific environment everywhere. Right? So if you really think about, if you had 1,000 different companies building automation, there was 1,000 different automations they were building. So it was 1,000 times the amount of work that you need to do.
Jim Walker:
Yeah.
Scott McCarty:
Then I would say by like 2005 to '10, even '11, '12, we had OpenStack and we started to see the writing on the wall. They're like, "Wait a minute. I can interact with the API. Then I've got to build the automation once and I can deploy the automation at the 1,000 sites, but write it once." We could see it, but it didn't quite go to the point where you could select the automation that you want and deploy. There's a chef in the kitchen that was still choosing the ingredients and figuring out how to make the food. But there wasn't a menu, right? There was no menu. You couldn't just order-
Jim Walker:
That's right.
Scott McCarty:
.... it was like cooking at your house. I saw a lot of projects fail because I was actually a solutions architect at the time for Red Hat. I remember adding one customer in particular. They had this wildly ambitious plan. They used ServiceNow. All this crazy homegrown automation with like Ansible, and Chef, and Puppet or whatever they were using. They had like all the moving parts of what's in the marketplace today, but there was no standard way to do the automation. It was still an open system that didn't have constraints placed on it. I'd argued now for the first time, since we've standardized on Kubernetes, we've kind of standardized on this operator framework. Which really is a framework of like how you do the automation. You now have the ability to standardize a marketplace. This is like options trading. Options are standardized, there's a set price and a set date. So you can trade them in Chicago, nobody knew that in 1776. Nobody was like, "Oh, we'll just trade these options. I'll buy something at a future time." We realized that if you set the standards, you can actually trade these things on the market.
Jim Walker:
Okay.
Scott McCarty:
That's what we finally done. We finally figured out how to get software into a market and trade it essentially for dollars.
Jim Walker:
Right.
Scott McCarty:
Because all the automation is standard. So now, if you go there and you look at CockroachDB, you click on a button, it'll install. You know it will install because it was built on specific versions of OpenShift.
Jim Walker:
That's right.
Scott McCarty:
So you know that will work. We're really close to the dream, if you will, the dream state.
Jim Walker:
Yeah.
Scott McCarty:
We're finally to the electricity where you just buy the electricity and it actually works.
Jim Walker:
That's right. That's right. I think it's just taken time. I've contended over the past five to 10 years, what we've done is we've basically abstracted out all the components of the software delivery supply chain. From the fingertips of the, of the developer all the way through to the mouse click of the user and everything that has to happen in between. We finally have codified that-
Scott McCarty:
Finally.
Jim Walker:
... into its constituent parts and it's all API based so that, that entire supply chain... Just as we've done in the past with a physical supply chain, like a wool jacket, like you're talking about. Right? The time it takes and all the components from the raw materials to packaging, to getting it in the store, putting a price in a skew and all that. We basically created a very well greased automated supply chain for software. I think Kubernetes is basically now expanding that to even deeper depths of global deployments. I think doing that has made huge changes. It wouldn't have been done without the likes of the companies like the Chefs and Puppet. Those were huge, massive steps. Everybody the CloudBees team and what they're doing in CI/CD a huge piece of this.
Jim Walker:
I think about LaunchDarkly and what they're doing with feature flagging. How cool is that, that we have that in the supply chain? I feel like OpenShift Scott, the culmination of a lot of those things. It's Ford versus the one-off car, right? That's where we're headed, right? It's like the full end to end production facility. Correct?
Scott McCarty:
Yeah. I liken it to like Swatch. People don't know, but in 2010 ish, they created the first automatic watch that was touchless. From the plastic to the time it rolled off the assembly line and you literally put in a box and everything. You just went in the truck and it went to the store. It was historically for like 500 years, we've been making watches or whatever it is, there always had to be a watchmaker. The closest they'd gotten it like into the 2000s was having a watchmaker fine tune it at the very end and they'd-
Jim Walker:
Right.
Scott McCarty:
But they figured out a laser way to do it now that it's like completely touchless manufacturing. What we've done is touchless manufacturing, but instead of just for one company for one product line, we've done it for an ecosystem of products.
Jim Walker:
That's right.
Scott McCarty:
Like now, CockroachDB, and even your competitors, and your friends, and allies, and all these different pieces of software, they get all built on the same essentially assembly line that basically allows them to have this touchless manufacturing.
Jim Walker:
the trick is, it's all going to end up in different stores and in different countries and all these different things. So I'll ask you a bit of a directive question. I'm a believer in this thing, but do we live in a multi-cloud world? Is that basically the future? I think there's some people who say, "No way, it's never going to happen. It's too complex." I'm a believer in it personally.
Scott McCarty:
Here's what nobody realizes, just like the cloud... Do you remember the narrative seven years ago when we were like, "You're already using Amazon and you don't know it."
Jim Walker:
Yeah, right.
Scott McCarty:
Well, you're already using Azure and Amazon and you don't know it like.
Jim Walker:
Yeah.
Scott McCarty:
The CIO doesn't know, but he has developers in this group doing this thing and developers in another group. I've seen CIO do stuff as crazy as not allow... Literally working with a credit card company to limit where people spend money. Because, they can't get a handle on it. They can't control where everybody's going. I would say, I don't know, 80% of the fortune 500 is probably already have multi-cloud and they don't know it. So now it's going to be well, five years from now when we finally realize and wake up, "Oh, well, you actually are using... wait, we're using how many? Seven different clouds." Let's figure out how to make all this stuff work together, right?
Jim Walker:
Right.
Scott McCarty:
We definitely need to get a handle on this and then control the costs and blah, blah, blah.
Jim Walker:
Well, I asked the right question. You just almost threw your microphone right off the table, dude. I mean, like Scott-
Scott McCarty:
I'm passionate about this one.
Jim Walker:
I'm super passionate about this one. In fact, we created a multi-cloud conference last year called Escape. I know the Red Hat team was extremely supportive of it because I think they shared the vision. Personally, I've been an OpenSource person for a while. Jim Whitehurst and Paul Cormier, the leaders of Red Hat have done such a phenomenal job, actually understanding that the world is not homogenous, right? The world is heterogeneity and it's always going to be that way. By the way, the world is also open.
Scott McCarty:
Yeah. Exactly.
Jim Walker:
I think that's a lot of the trick. I commend you guys all y'all at Red Hat. The delivery of OpenShift and what I'm seeing an OpenShift 4, and I'm sorry, but I'm an old CoreOS person. I’m really proud to see this really coming to light. To see the marketplace in its reality. Looking back to my conversation with Brandon Phillips a long time ago, "Like, man, it'd be great if you could just push a button and deploy." By the way, not even worry about upgrades, because that's just all automated and background.
Scott McCarty:
Yep.
Jim Walker:
To me the-
Scott McCarty:
It's the app store of B2B apps-
Jim Walker:
That's right.
Scott McCarty:
... as opposed to just a B2C app on your phone.
Jim Walker:
That's right. Kudos to the team on everything that's been going on. But we're at the top of the hour. Scott, thank you so much. That was a great conversation. I had a hard time getting a word in edgewise, but that's good. It was really good. Sorry, Tim-
Tim:
Hey, I can just as easily be quiet on these things.I liked it too. It was great. It was awesome. So thank you.
Jim Walker:
Yeah. Super insightful. So, but seriously, thank you. Thanks for joining us, Scott. It was really, really a lot of fun. Say, Hey to Rob from me over there.
Scott McCarty:
I will. Thank you for having me. I appreciate it.
Jim Walker:
Alrighty. Everybody, yeah. Recording this will be available. There are no slides for you to send. But I hope it was useful for everybody. I think I believe we, we still send out a survey, so please do fill out the survey. Let us know how we did. We're always trying to improve these things as always, it's constantly evolving. So for on behalf of Three Guys with Flannel Shirts, thank you everybody for joining us today. So thank you everybody for joining us and I hope you have an enjoyable rest of the day. Bye now.
